<!-- build time:Fri Nov 18 2016 13:02:35 GMT+0800 (CST) --><!DOCTYPE HTML><html lang="zh-CN"><head><meta charset="UTF-8"><title>动态加载的页面数据的抓取-爱红旗渠</title><meta name="viewport" content="width=device-width,initial-scale=1,user-scalable=no"><meta name="author" content="Jin"><meta name="keywords" content="动态页面数据抓取，动态加载的页面数据的抓取，动态数据抓取，抓取ajax渲染的页面，, 爱红旗渠, ihongqiqu, ihongqiqu.com, aihongqiqu"><meta name="description" content="&amp;lt;!-- build time:Fri Nov 18 2016 13:02:32 GMT+0800 (CST) --&amp;gt;前些天写了《初识网络爬虫》，一般的网络爬虫是通过获取源码，并从源码中获取数据。问题来了，现在 ajax 技术应用也很普遍，针对如此情况，我们一般的爬虫无法获取到需要的数据。本文主要实现抓取动态页面的数据。"><meta property="og:type" content="article"><meta property="og:title" content="动态加载的页面数据的抓取"><meta property="og:url" content="http://ihongqiqu.com/2016/11/07/web-crawler-2/index.html"><meta property="og:site_name" content="爱红旗渠"><meta property="og:description" content="&amp;lt;!-- build time:Fri Nov 18 2016 13:02:32 GMT+0800 (CST) --&amp;gt;前些天写了《初识网络爬虫》，一般的网络爬虫是通过获取源码，并从源码中获取数据。问题来了，现在 ajax 技术应用也很普遍，针对如此情况，我们一般的爬虫无法获取到需要的数据。本文主要实现抓取动态页面的数据。"><meta property="og:updated_time" content="2016-11-06T17:16:51.000Z"><meta name="twitter:card" content="summary"><meta name="twitter:title" content="动态加载的页面数据的抓取"><meta name="twitter:description" content="&amp;lt;!-- build time:Fri Nov 18 2016 13:02:32 GMT+0800 (CST) --&amp;gt;前些天写了《初识网络爬虫》，一般的网络爬虫是通过获取源码，并从源码中获取数据。问题来了，现在 ajax 技术应用也很普遍，针对如此情况，我们一般的爬虫无法获取到需要的数据。本文主要实现抓取动态页面的数据。"><link rel="alternative" href="/atom.xml" title="爱红旗渠" type="application/atom+xml"><link rel="icon" href="/favicon.ico"><link rel="apple-touch-icon" href="/imgs/myself.jpg"><link rel="apple-touch-icon-precomposed" href="/imgs/myself.jpg"><link rel="stylesheet" href="/css/style.css" type="text/css"></head></html><body><header><div><div id="textlogo"><h1 class="site-name"><a href="/" title="爱红旗渠">爱红旗渠</a></h1><h2 class="blog-motto">Stay hungry, stay foolish.</h2></div><div class="navbar"><a class="navbutton navmobile" href="#" title="菜单"></a></div><nav class="animated"><ul><ul><li><a href="/">首页</a></li><li><a href="/archives">归档</a></li><li><a href="/about">关于</a></li><li><form class="search" action="/search/index.html" method="get" accept-charset="utf-8"><label>Search</label><input type="text" id="search" class="st-default-search-input" maxlength="20" placeholder="Search"></form></li></ul></ul></nav></div></header><div id="container"><div id="main" class="post" itemscope itemprop="blogPost"><article itemprop="articleBody"><header class="article-info clearfix"><h1 itemprop="name"><a href="/2016/11/07/web-crawler-2/" title="动态加载的页面数据的抓取" itemprop="url">动态加载的页面数据的抓取</a></h1><p class="article-author">By <a href="/about" title="Jin" target="_blank" itemprop="author">Jin</a></p><p class="article-time"><time datetime="2016-11-06T17:01:02.000Z" itemprop="datePublished">发表于 2016-11-07</time></p></header><div class="article-content"><div id="toc" class="toc-article"><strong class="toc-title">文章目录</strong><ol class="toc"><li class="toc-item toc-level-3"><a class="toc-link" href="#u5206_u6790"><span class="toc-number">1.</span> <span class="toc-text">分析</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#u5B9E_u73B0"><span class="toc-number">2.</span> <span class="toc-text">实现</span></a><ol class="toc-child"><li class="toc-item toc-level-4"><a class="toc-link" href="#u6DFB_u52A0_u4F9D_u8D56"><span class="toc-number">2.1.</span> <span class="toc-text">添加依赖</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u4E0B_u8F7D_driver"><span class="toc-number">2.2.</span> <span class="toc-text">下载 driver</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u5B89_u88C5_Chrome__u6D4F_u89C8_u5668"><span class="toc-number">2.3.</span> <span class="toc-text">安装 Chrome 浏览器</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u6D4B_u8BD5"><span class="toc-number">2.4.</span> <span class="toc-text">测试</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u5B9E_u73B0_u722C_u866B"><span class="toc-number">2.5.</span> <span class="toc-text">实现爬虫</span></a></li></ol></li></ol></div><p>前些天写了《<a href="http://ihongqiqu.com/2016/10/26/web-crawler/">初识网络爬虫</a>》，一般的网络爬虫是通过获取源码，并从源码中获取数据。问题来了，现在 ajax 技术应用也很普遍，针对如此情况，我们一般的爬虫无法获取到需要的数据。本文主要实现抓取动态页面的数据。</p><a id="more"></a><p>动态加载页面数据有两种方法可以选择：</p><ol><li>模拟页面中的请求，直接获取接口返回的数据</li><li>内建浏览器渲染页面，然后获取渲染后的数据</li></ol><h3 id="u5206_u6790"><a href="#u5206_u6790" class="headerlink" title="分析"></a>分析</h3><p>在页面中通过拼凑参数等方法来模拟网络请求，最终获取接口数据，这种方法是可以行的通的，问题是比较麻烦。本文主要通过内建浏览器渲染这种简单粗暴的方法来实现数据的抓取。</p><p>问题来了，如何内建浏览器呢？</p><p>熟悉自动化测试同学应该都知道 <a href="http://docs.seleniumhq.org/projects/" rel="external nofollow" target="_blank">Selenium</a>，这个模拟浏览器进行自动化测试的工具。Selenium 提供一组 API 可以与真实的浏览器内核交互。Selenium 是跨语言的，有 Java、C#、python 等版本，并且支持多种浏览器，chrome、firefox 以及 IE 都支持。</p><h3 id="u5B9E_u73B0"><a href="#u5B9E_u73B0" class="headerlink" title="实现"></a>实现</h3><p>我们用 Java 来写 Demo。</p><h4 id="u6DFB_u52A0_u4F9D_u8D56"><a href="#u6DFB_u52A0_u4F9D_u8D56" class="headerlink" title="添加依赖"></a>添加依赖</h4><p>添加 Selenium 依赖，以 Maven 为例：</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="title">dependency</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">groupId</span>&gt;</span>org.seleniumhq.selenium<span class="tag">&lt;/<span class="title">groupId</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">artifactId</span>&gt;</span>selenium-java<span class="tag">&lt;/<span class="title">artifactId</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">version</span>&gt;</span>3.0.1<span class="tag">&lt;/<span class="title">version</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="title">dependency</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="title">dependency</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">groupId</span>&gt;</span>org.seleniumhq.selenium<span class="tag">&lt;/<span class="title">groupId</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">artifactId</span>&gt;</span>selenium-chrome-driver<span class="tag">&lt;/<span class="title">artifactId</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">version</span>&gt;</span>3.0.1<span class="tag">&lt;/<span class="title">version</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="title">dependency</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="title">dependency</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">groupId</span>&gt;</span>org.seleniumhq.selenium<span class="tag">&lt;/<span class="title">groupId</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">artifactId</span>&gt;</span>selenium-server<span class="tag">&lt;/<span class="title">artifactId</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="title">version</span>&gt;</span>2.18.0<span class="tag">&lt;/<span class="title">version</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="title">dependency</span>&gt;</span></span><br></pre></td></tr></table></figure><h4 id="u4E0B_u8F7D_driver"><a href="#u4E0B_u8F7D_driver" class="headerlink" title="下载 driver"></a>下载 driver</h4><p>以 chrome 为例：<a href="https://sites.google.com/a/chromium.org/chromedriver/" rel="external nofollow" target="_blank">https://sites.google.com/a/chromium.org/chromedriver/</a></p><p>下载后，最好添加环境变量。当然，也可以在调用前设置环境:</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">System.getProperties().setProperty(<span class="string">"webdriver.chrome.driver"</span>,</span><br><span class="line">    <span class="string">"/Users/zhenguo/Documents/chrome/chromedriver"</span>);</span><br></pre></td></tr></table></figure><p>注意：MAC 环境下需要确认 chromedriver 是可运行的。</p><h4 id="u5B89_u88C5_Chrome__u6D4F_u89C8_u5668"><a href="#u5B89_u88C5_Chrome__u6D4F_u89C8_u5668" class="headerlink" title="安装 Chrome 浏览器"></a>安装 Chrome 浏览器</h4><p>这个，没什么好说的。</p><h4 id="u6D4B_u8BD5"><a href="#u6D4B_u8BD5" class="headerlink" title="测试"></a>测试</h4><p>测试 selenium ，代码如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="annotation">@Ignore</span>(<span class="string">"need chrome driver"</span>)</span><br><span class="line"><span class="annotation">@Test</span></span><br><span class="line"><span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">testSelenium</span><span class="params">()</span> </span>&#123;</span><br><span class="line">    System.getProperties().setProperty(<span class="string">"webdriver.chrome.driver"</span>,</span><br><span class="line">        <span class="string">"/Users/zhenguo/Documents/chrome/chromedriver"</span>);</span><br><span class="line">    WebDriver webDriver = <span class="keyword">new</span> ChromeDriver();</span><br><span class="line">    webDriver.get(<span class="string">"http://huaban.com/"</span>);</span><br><span class="line">    WebElement webElement = webDriver.findElement(By.xpath(<span class="string">"/html"</span>));</span><br><span class="line">    System.out.println(webElement.getAttribute(<span class="string">"outerHTML"</span>));</span><br><span class="line">    webDriver.close();</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>如果出现类似以下结果，就说明 webdriver 配置好了：</p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">Starting ChromeDriver 2.25.426935 (820a95b0b81d33e42712f9198c215f703412e1a1) on port 2052</span><br><span class="line">Only local connections are allowed.</span><br><span class="line">Nov 07, 2016 12:35:11 AM org.openqa.selenium.remote.ProtocolHandshake createSession</span><br><span class="line">INFO: Attempting bi-dialect session, assuming Postel's Law holds true on the remote end</span><br><span class="line">Nov 07, 2016 12:35:13 AM org.openqa.selenium.remote.ProtocolHandshake createSession</span><br><span class="line">INFO: Detected dialect: OSS</span><br></pre></td></tr></table></figure><p>PS：每次<code>new ChromeDriver()</code>，Selenium都会建立一个Chrome进程，并使用一个随机端口在Java中与chrome进程进行通信来交互。我们需要调用 <code>webDriver.close()</code> 关闭进程。如果是网络爬虫抓取数据的话，最好用线程池来处理。</p><h4 id="u5B9E_u73B0_u722C_u866B"><a href="#u5B9E_u73B0_u722C_u866B" class="headerlink" title="实现爬虫"></a>实现爬虫</h4><p>上面步骤都设置好了，基于 webmagic 的爬虫实现就比较简单了，代码如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">HuabanProcessor</span> <span class="keyword">implements</span> <span class="title">PageProcessor</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> Site site;</span><br><span class="line"></span><br><span class="line">    <span class="annotation">@Override</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">process</span><span class="params">(Page page)</span> </span>&#123;</span><br><span class="line">           page.addTargetRequests(</span><br><span class="line">             page.getHtml().links().regex(<span class="string">"http://huaban\\.com/.*"</span>).all());</span><br><span class="line">        <span class="keyword">if</span> (page.getUrl().toString().contains(<span class="string">"pins"</span>)) &#123;</span><br><span class="line">            page.putField(<span class="string">"img"</span>, page.getHtml().</span><br><span class="line">                          xpath(<span class="string">"//div[@id='baidu_image_holder']/img/@src"</span>).toString());</span><br><span class="line">        &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">            page.getResultItems().setSkip(<span class="keyword">true</span>);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="annotation">@Override</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> Site <span class="title">getSite</span><span class="params">()</span> </span>&#123;</span><br><span class="line">        <span class="keyword">if</span> (<span class="keyword">null</span> == site) &#123;</span><br><span class="line">            site = Site.me().setDomain(<span class="string">"huaban.com"</span>).setSleepTime(<span class="number">0</span>);</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> site;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">static</span> <span class="keyword">void</span> <span class="title">main</span><span class="params">(String[] args)</span> </span>&#123;</span><br><span class="line">        Spider.create(<span class="keyword">new</span> HuabanProcessor()).thread(<span class="number">5</span>)</span><br><span class="line">                .addPipeline(<span class="keyword">new</span> FilePipeline(<span class="string">"/Users/zhenguo/Documents/chrome/webmagic/test/"</span>))</span><br><span class="line">                .setDownloader(<span class="keyword">new</span> SeleniumDownloader(<span class="string">"/Users/zhenguo/Documents/chrome/chromedriver"</span>))</span><br><span class="line">                .addUrl(<span class="string">"http://huaban.com/explore/gufenghaibao/"</span>)</span><br><span class="line">                .runAsync();</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>上面 <code>HuabanProcessor</code> 使用到 <code>SeleniumDownloader</code>，代码如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br></pre></td><td class="code"><pre><span class="line">   <span class="keyword">package</span> us.codecraft.webmagic.downloader.selenium;</span><br><span class="line"></span><br><span class="line">   <span class="keyword">import</span> org.apache.log4j.Logger;</span><br><span class="line">   <span class="keyword">import</span> org.openqa.selenium.By;</span><br><span class="line">   <span class="keyword">import</span> org.openqa.selenium.Cookie;</span><br><span class="line">   <span class="keyword">import</span> org.openqa.selenium.WebDriver;</span><br><span class="line">   <span class="keyword">import</span> org.openqa.selenium.WebElement;</span><br><span class="line"></span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.Page;</span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.Request;</span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.Site;</span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.Task;</span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.downloader.Downloader;</span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.selector.Html;</span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.selector.PlainText;</span><br><span class="line">   <span class="keyword">import</span> us.codecraft.webmagic.utils.UrlUtils;</span><br><span class="line"></span><br><span class="line">   <span class="keyword">import</span> java.io.Closeable;</span><br><span class="line">   <span class="keyword">import</span> java.io.IOException;</span><br><span class="line">   <span class="keyword">import</span> java.util.Map;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span><br><span class="line"> * 使用Selenium调用浏览器进行渲染。目前仅支持chrome。&lt;br&gt;</span><br><span class="line"> * 需要下载Selenium driver支持。&lt;br&gt;</span><br><span class="line"> *</span><br><span class="line"> * <span class="doctag">@author</span> code4crafter<span class="doctag">@gmail</span>.com &lt;br&gt;</span><br><span class="line"> *         Date: 13-7-26 &lt;br&gt;</span><br><span class="line"> *         Time: 下午1:37 &lt;br&gt;</span><br><span class="line"> */</span></span><br><span class="line"> <span class="keyword">public</span> <span class="class"><span class="keyword">class</span> <span class="title">SeleniumDownloader</span> <span class="keyword">implements</span> <span class="title">Downloader</span>, <span class="title">Closeable</span> </span>&#123;</span><br><span class="line"></span><br><span class="line">     <span class="keyword">private</span> <span class="keyword">volatile</span> WebDriverPool webDriverPool;</span><br><span class="line"></span><br><span class="line">     <span class="keyword">private</span> Logger logger = Logger.getLogger(getClass());</span><br><span class="line"></span><br><span class="line">     <span class="keyword">private</span> <span class="keyword">int</span> sleepTime = <span class="number">0</span>;</span><br><span class="line"></span><br><span class="line">     <span class="keyword">private</span> <span class="keyword">int</span> poolSize = <span class="number">1</span>;</span><br><span class="line"></span><br><span class="line">     <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String DRIVER_PHANTOMJS = <span class="string">"phantomjs"</span>;</span><br><span class="line"></span><br><span class="line">     <span class="comment">/**</span><br><span class="line">      * 新建</span><br><span class="line">      *</span><br><span class="line">      * <span class="doctag">@param</span> chromeDriverPath chromeDriverPath</span><br><span class="line">      */</span></span><br><span class="line">     <span class="function"><span class="keyword">public</span> <span class="title">SeleniumDownloader</span><span class="params">(String chromeDriverPath)</span> </span>&#123;</span><br><span class="line">         System.getProperties().setProperty(<span class="string">"webdriver.chrome.driver"</span>,</span><br><span class="line">                 chromeDriverPath);</span><br><span class="line">     &#125;</span><br><span class="line"></span><br><span class="line">     <span class="comment">/**</span><br><span class="line">      * Constructor without any filed. Construct PhantomJS browser</span><br><span class="line">      * </span><br><span class="line">      * <span class="doctag">@author</span> bob.li.0718<span class="doctag">@gmail</span>.com</span><br><span class="line">      */</span></span><br><span class="line">     <span class="function"><span class="keyword">public</span> <span class="title">SeleniumDownloader</span><span class="params">()</span> </span>&#123;</span><br><span class="line">         <span class="comment">// System.setProperty("phantomjs.binary.path",</span></span><br><span class="line">         <span class="comment">// "/Users/Bingo/Downloads/phantomjs-1.9.7-macosx/bin/phantomjs");</span></span><br><span class="line">     &#125;</span><br><span class="line"></span><br><span class="line">     <span class="comment">/**</span><br><span class="line">      * set sleep time to wait until load success</span><br><span class="line">      *</span><br><span class="line">      * <span class="doctag">@param</span> sleepTime sleepTime</span><br><span class="line">      * <span class="doctag">@return</span> this</span><br><span class="line">      */</span></span><br><span class="line">     <span class="function"><span class="keyword">public</span> SeleniumDownloader <span class="title">setSleepTime</span><span class="params">(<span class="keyword">int</span> sleepTime)</span> </span>&#123;</span><br><span class="line">         <span class="keyword">this</span>.sleepTime = sleepTime;</span><br><span class="line">         <span class="keyword">return</span> <span class="keyword">this</span>;</span><br><span class="line">     &#125;</span><br><span class="line"></span><br><span class="line">     <span class="annotation">@Override</span></span><br><span class="line">     <span class="function"><span class="keyword">public</span> Page <span class="title">download</span><span class="params">(Request request, Task task)</span> </span>&#123;</span><br><span class="line">         checkInit();</span><br><span class="line">         WebDriver webDriver;</span><br><span class="line">         <span class="keyword">try</span> &#123;</span><br><span class="line">             webDriver = webDriverPool.get();</span><br><span class="line">         &#125; <span class="keyword">catch</span> (InterruptedException e) &#123;</span><br><span class="line">             logger.warn(<span class="string">"interrupted"</span>, e);</span><br><span class="line">             <span class="keyword">return</span> <span class="keyword">null</span>;</span><br><span class="line">         &#125;</span><br><span class="line">         logger.info(<span class="string">"downloading page "</span> + request.getUrl());</span><br><span class="line">         webDriver.get(request.getUrl());</span><br><span class="line">         <span class="keyword">try</span> &#123;</span><br><span class="line">             Thread.sleep(sleepTime);</span><br><span class="line">         &#125; <span class="keyword">catch</span> (InterruptedException e) &#123;</span><br><span class="line">             e.printStackTrace();</span><br><span class="line">         &#125;</span><br><span class="line">         WebDriver.Options manage = webDriver.manage();</span><br><span class="line">         Site site = task.getSite();</span><br><span class="line">         <span class="keyword">if</span> (site.getCookies() != <span class="keyword">null</span>) &#123;</span><br><span class="line">             <span class="keyword">for</span> (Map.Entry&lt;String, String&gt; cookieEntry : site.getCookies()</span><br><span class="line">                     .entrySet()) &#123;</span><br><span class="line">                 Cookie cookie = <span class="keyword">new</span> Cookie(cookieEntry.getKey(),</span><br><span class="line">                         cookieEntry.getValue());</span><br><span class="line">                 manage.addCookie(cookie);</span><br><span class="line">             &#125;</span><br><span class="line">         &#125;</span><br><span class="line"></span><br><span class="line">         <span class="comment">/*</span><br><span class="line">          * TODO You can add mouse event or other processes</span><br><span class="line">          * </span><br><span class="line">          * @author: bob.li.0718@gmail.com</span><br><span class="line">          */</span></span><br><span class="line"></span><br><span class="line">         WebElement webElement = webDriver.findElement(By.xpath(<span class="string">"/html"</span>));</span><br><span class="line">         String content = webElement.getAttribute(<span class="string">"outerHTML"</span>);</span><br><span class="line">         Page page = <span class="keyword">new</span> Page();</span><br><span class="line">         page.setRawText(content);</span><br><span class="line">         page.setHtml(<span class="keyword">new</span> Html(UrlUtils.fixAllRelativeHrefs(content,</span><br><span class="line">                 request.getUrl())));</span><br><span class="line">         page.setUrl(<span class="keyword">new</span> PlainText(request.getUrl()));</span><br><span class="line">         page.setRequest(request);</span><br><span class="line">         webDriverPool.returnToPool(webDriver);</span><br><span class="line">         <span class="keyword">return</span> page;</span><br><span class="line">     &#125;</span><br><span class="line"></span><br><span class="line">     <span class="function"><span class="keyword">private</span> <span class="keyword">void</span> <span class="title">checkInit</span><span class="params">()</span> </span>&#123;</span><br><span class="line">         <span class="keyword">if</span> (webDriverPool == <span class="keyword">null</span>) &#123;</span><br><span class="line">             <span class="keyword">synchronized</span> (<span class="keyword">this</span>) &#123;</span><br><span class="line">                 webDriverPool = <span class="keyword">new</span> WebDriverPool(poolSize);</span><br><span class="line">             &#125;</span><br><span class="line">         &#125;</span><br><span class="line">     &#125;</span><br><span class="line"></span><br><span class="line">     <span class="annotation">@Override</span></span><br><span class="line">     <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">setThread</span><span class="params">(<span class="keyword">int</span> thread)</span> </span>&#123;</span><br><span class="line">         <span class="keyword">this</span>.poolSize = thread;</span><br><span class="line">     &#125;</span><br><span class="line"></span><br><span class="line">     <span class="annotation">@Override</span></span><br><span class="line">     <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">close</span><span class="params">()</span> <span class="keyword">throws</span> IOException </span>&#123;</span><br><span class="line">         webDriverPool.closeAll();</span><br><span class="line">     &#125;</span><br><span class="line"> &#125;</span><br></pre></td></tr></table></figure><p><code>WebDriverPool</code> 代码如下：</p><figure class="highlight java"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> us.codecraft.webmagic.downloader.selenium;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> org.apache.log4j.Logger;</span><br><span class="line"><span class="keyword">import</span> org.openqa.selenium.WebDriver;</span><br><span class="line"><span class="keyword">import</span> org.openqa.selenium.chrome.ChromeDriver;</span><br><span class="line"><span class="keyword">import</span> org.openqa.selenium.firefox.FirefoxDriver;</span><br><span class="line"><span class="keyword">import</span> org.openqa.selenium.phantomjs.PhantomJSDriver;</span><br><span class="line"><span class="keyword">import</span> org.openqa.selenium.phantomjs.PhantomJSDriverService;</span><br><span class="line"><span class="keyword">import</span> org.openqa.selenium.remote.DesiredCapabilities;</span><br><span class="line"><span class="keyword">import</span> org.openqa.selenium.remote.RemoteWebDriver;</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> java.io.FileReader;</span><br><span class="line"><span class="keyword">import</span> java.io.IOException;</span><br><span class="line"><span class="keyword">import</span> java.net.MalformedURLException;</span><br><span class="line"><span class="keyword">import</span> java.net.URL;</span><br><span class="line"><span class="keyword">import</span> java.util.ArrayList;</span><br><span class="line"><span class="keyword">import</span> java.util.Collections;</span><br><span class="line"><span class="keyword">import</span> java.util.List;</span><br><span class="line"><span class="keyword">import</span> java.util.Properties;</span><br><span class="line"><span class="keyword">import</span> java.util.concurrent.BlockingDeque;</span><br><span class="line"><span class="keyword">import</span> java.util.concurrent.LinkedBlockingDeque;</span><br><span class="line"><span class="keyword">import</span> java.util.concurrent.atomic.AtomicInteger;</span><br><span class="line"></span><br><span class="line"><span class="comment">/**</span><br><span class="line"> * <span class="doctag">@author</span> code4crafter<span class="doctag">@gmail</span>.com &lt;br&gt;</span><br><span class="line"> *         Date: 13-7-26 &lt;br&gt;</span><br><span class="line"> *         Time: 下午1:41 &lt;br&gt;</span><br><span class="line"> */</span></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">WebDriverPool</span> </span>&#123;</span><br><span class="line">    <span class="keyword">private</span> Logger logger = Logger.getLogger(getClass());</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">static</span> <span class="keyword">int</span> DEFAULT_CAPACITY = <span class="number">5</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">int</span> capacity;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">static</span> <span class="keyword">int</span> STAT_RUNNING = <span class="number">1</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">final</span> <span class="keyword">static</span> <span class="keyword">int</span> STAT_CLODED = <span class="number">2</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> AtomicInteger stat = <span class="keyword">new</span> AtomicInteger(STAT_RUNNING);</span><br><span class="line"></span><br><span class="line">    <span class="comment">/*</span><br><span class="line">     * new fields for configuring phantomJS</span><br><span class="line">     */</span></span><br><span class="line">    <span class="keyword">private</span> WebDriver mDriver = <span class="keyword">null</span>;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">boolean</span> mAutoQuitDriver = <span class="keyword">true</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String CONFIG_FILE = <span class="string">"/Users/zhenguo/Documents/develop/github/webmagic/webmagic-selenium/config.ini"</span>;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String DRIVER_FIREFOX = <span class="string">"firefox"</span>;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String DRIVER_CHROME = <span class="string">"chrome"</span>;</span><br><span class="line">    <span class="keyword">private</span> <span class="keyword">static</span> <span class="keyword">final</span> String DRIVER_PHANTOMJS = <span class="string">"phantomjs"</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">protected</span> <span class="keyword">static</span> Properties sConfig;</span><br><span class="line">    <span class="keyword">protected</span> <span class="keyword">static</span> DesiredCapabilities sCaps;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span><br><span class="line">     * Configure the GhostDriver, and initialize a WebDriver instance. This part</span><br><span class="line">     * of code comes from GhostDriver.</span><br><span class="line">     * https://github.com/detro/ghostdriver/tree/master/test/java/src/test/java/ghostdriver</span><br><span class="line">     * </span><br><span class="line">     * <span class="doctag">@author</span> bob.li.0718<span class="doctag">@gmail</span>.com</span><br><span class="line">     * <span class="doctag">@throws</span> IOException</span><br><span class="line">     */</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">configure</span><span class="params">()</span> <span class="keyword">throws</span> IOException </span>&#123;</span><br><span class="line">        <span class="comment">// Read config file</span></span><br><span class="line">        sConfig = <span class="keyword">new</span> Properties();</span><br><span class="line">        sConfig.load(<span class="keyword">new</span> FileReader(CONFIG_FILE));</span><br><span class="line"></span><br><span class="line">        <span class="comment">// Prepare capabilities</span></span><br><span class="line">        sCaps = <span class="keyword">new</span> DesiredCapabilities();</span><br><span class="line">        sCaps.setJavascriptEnabled(<span class="keyword">true</span>);</span><br><span class="line">        sCaps.setCapability(<span class="string">"takesScreenshot"</span>, <span class="keyword">false</span>);</span><br><span class="line"></span><br><span class="line">        String driver = sConfig.getProperty(<span class="string">"driver"</span>, DRIVER_PHANTOMJS);</span><br><span class="line"></span><br><span class="line">        <span class="comment">// Fetch PhantomJS-specific configuration parameters</span></span><br><span class="line">        <span class="keyword">if</span> (driver.equals(DRIVER_PHANTOMJS)) &#123;</span><br><span class="line">            <span class="comment">// "phantomjs_exec_path"</span></span><br><span class="line">            <span class="keyword">if</span> (sConfig.getProperty(<span class="string">"phantomjs_exec_path"</span>) != <span class="keyword">null</span>) &#123;</span><br><span class="line">                sCaps.setCapability(</span><br><span class="line">                        PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY,</span><br><span class="line">                        sConfig.getProperty(<span class="string">"phantomjs_exec_path"</span>));</span><br><span class="line">            &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">                <span class="keyword">throw</span> <span class="keyword">new</span> IOException(</span><br><span class="line">                        String.format(</span><br><span class="line">                                <span class="string">"Property '%s' not set!"</span>,</span><br><span class="line">                                PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY));</span><br><span class="line">            &#125;</span><br><span class="line">            <span class="comment">// "phantomjs_driver_path"</span></span><br><span class="line">            <span class="keyword">if</span> (sConfig.getProperty(<span class="string">"phantomjs_driver_path"</span>) != <span class="keyword">null</span>) &#123;</span><br><span class="line">                System.out.println(<span class="string">"Test will use an external GhostDriver"</span>);</span><br><span class="line">                sCaps.setCapability(</span><br><span class="line">                        PhantomJSDriverService.PHANTOMJS_GHOSTDRIVER_PATH_PROPERTY,</span><br><span class="line">                        sConfig.getProperty(<span class="string">"phantomjs_driver_path"</span>));</span><br><span class="line">            &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">                System.out</span><br><span class="line">                        .println(<span class="string">"Test will use PhantomJS internal GhostDriver"</span>);</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="comment">// Disable "web-security", enable all possible "ssl-protocols" and</span></span><br><span class="line">        <span class="comment">// "ignore-ssl-errors" for PhantomJSDriver</span></span><br><span class="line">        <span class="comment">// sCaps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS, new</span></span><br><span class="line">        <span class="comment">// String[] &#123;</span></span><br><span class="line">        <span class="comment">// "--web-security=false",</span></span><br><span class="line">        <span class="comment">// "--ssl-protocol=any",</span></span><br><span class="line">        <span class="comment">// "--ignore-ssl-errors=true"</span></span><br><span class="line">        <span class="comment">// &#125;);</span></span><br><span class="line"></span><br><span class="line">        ArrayList&lt;String&gt; cliArgsCap = <span class="keyword">new</span> ArrayList&lt;String&gt;();</span><br><span class="line">        cliArgsCap.add(<span class="string">"--web-security=false"</span>);</span><br><span class="line">        cliArgsCap.add(<span class="string">"--ssl-protocol=any"</span>);</span><br><span class="line">        cliArgsCap.add(<span class="string">"--ignore-ssl-errors=true"</span>);</span><br><span class="line">        sCaps.setCapability(PhantomJSDriverService.PHANTOMJS_CLI_ARGS,</span><br><span class="line">                cliArgsCap);</span><br><span class="line"></span><br><span class="line">        <span class="comment">// Control LogLevel for GhostDriver, via CLI arguments</span></span><br><span class="line">        sCaps.setCapability(</span><br><span class="line">                PhantomJSDriverService.PHANTOMJS_GHOSTDRIVER_CLI_ARGS,</span><br><span class="line">                <span class="keyword">new</span> String[] &#123; <span class="string">"--logLevel="</span></span><br><span class="line">                        + (sConfig.getProperty(<span class="string">"phantomjs_driver_loglevel"</span>) != <span class="keyword">null</span> ? sConfig</span><br><span class="line">                                .getProperty(<span class="string">"phantomjs_driver_loglevel"</span>)</span><br><span class="line">                                : <span class="string">"INFO"</span>) &#125;);</span><br><span class="line"></span><br><span class="line">        <span class="comment">// String driver = sConfig.getProperty("driver", DRIVER_PHANTOMJS);</span></span><br><span class="line"></span><br><span class="line">        <span class="comment">// Start appropriate Driver</span></span><br><span class="line">        <span class="keyword">if</span> (isUrl(driver)) &#123;</span><br><span class="line">            sCaps.setBrowserName(<span class="string">"phantomjs"</span>);</span><br><span class="line">            mDriver = <span class="keyword">new</span> RemoteWebDriver(<span class="keyword">new</span> URL(driver), sCaps);</span><br><span class="line">        &#125; <span class="keyword">else</span> <span class="keyword">if</span> (driver.equals(DRIVER_FIREFOX)) &#123;</span><br><span class="line">            mDriver = <span class="keyword">new</span> FirefoxDriver(sCaps);</span><br><span class="line">        &#125; <span class="keyword">else</span> <span class="keyword">if</span> (driver.equals(DRIVER_CHROME)) &#123;</span><br><span class="line">            mDriver = <span class="keyword">new</span> ChromeDriver(sCaps);</span><br><span class="line">        &#125; <span class="keyword">else</span> <span class="keyword">if</span> (driver.equals(DRIVER_PHANTOMJS)) &#123;</span><br><span class="line">            mDriver = <span class="keyword">new</span> PhantomJSDriver(sCaps);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span><br><span class="line">     * check whether input is a valid URL</span><br><span class="line">     * </span><br><span class="line">     * <span class="doctag">@author</span> bob.li.0718<span class="doctag">@gmail</span>.com</span><br><span class="line">     * <span class="doctag">@param</span> urlString urlString</span><br><span class="line">     * <span class="doctag">@return</span> true means yes, otherwise no.</span><br><span class="line">     */</span></span><br><span class="line">    <span class="function"><span class="keyword">private</span> <span class="keyword">boolean</span> <span class="title">isUrl</span><span class="params">(String urlString)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">try</span> &#123;</span><br><span class="line">            <span class="keyword">new</span> URL(urlString);</span><br><span class="line">            <span class="keyword">return</span> <span class="keyword">true</span>;</span><br><span class="line">        &#125; <span class="keyword">catch</span> (MalformedURLException mue) &#123;</span><br><span class="line">            <span class="keyword">return</span> <span class="keyword">false</span>;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span><br><span class="line">     * store webDrivers created</span><br><span class="line">     */</span></span><br><span class="line">    <span class="keyword">private</span> List&lt;WebDriver&gt; webDriverList = Collections</span><br><span class="line">            .synchronizedList(<span class="keyword">new</span> ArrayList&lt;WebDriver&gt;());</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span><br><span class="line">     * store webDrivers available</span><br><span class="line">     */</span></span><br><span class="line">    <span class="keyword">private</span> BlockingDeque&lt;WebDriver&gt; innerQueue = <span class="keyword">new</span> LinkedBlockingDeque&lt;WebDriver&gt;();</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="title">WebDriverPool</span><span class="params">(<span class="keyword">int</span> capacity)</span> </span>&#123;</span><br><span class="line">        <span class="keyword">this</span>.capacity = capacity;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="title">WebDriverPool</span><span class="params">()</span> </span>&#123;</span><br><span class="line">        <span class="keyword">this</span>(DEFAULT_CAPACITY);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">/**</span><br><span class="line">     * </span><br><span class="line">     * <span class="doctag">@return</span></span><br><span class="line">     * <span class="doctag">@throws</span> InterruptedException</span><br><span class="line">     */</span></span><br><span class="line">    <span class="function"><span class="keyword">public</span> WebDriver <span class="title">get</span><span class="params">()</span> <span class="keyword">throws</span> InterruptedException </span>&#123;</span><br><span class="line">        checkRunning();</span><br><span class="line">        WebDriver poll = innerQueue.poll();</span><br><span class="line">        <span class="keyword">if</span> (poll != <span class="keyword">null</span>) &#123;</span><br><span class="line">            <span class="keyword">return</span> poll;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">if</span> (webDriverList.size() &lt; capacity) &#123;</span><br><span class="line">            <span class="keyword">synchronized</span> (webDriverList) &#123;</span><br><span class="line">                <span class="keyword">if</span> (webDriverList.size() &lt; capacity) &#123;</span><br><span class="line"></span><br><span class="line">                    <span class="comment">// add new WebDriver instance into pool</span></span><br><span class="line">                    <span class="keyword">try</span> &#123;</span><br><span class="line">                        configure();</span><br><span class="line">                        innerQueue.add(mDriver);</span><br><span class="line">                        webDriverList.add(mDriver);</span><br><span class="line">                    &#125; <span class="keyword">catch</span> (IOException e) &#123;</span><br><span class="line">                        e.printStackTrace();</span><br><span class="line">                    &#125;</span><br><span class="line"></span><br><span class="line">                    <span class="comment">// ChromeDriver e = new ChromeDriver();</span></span><br><span class="line">                    <span class="comment">// WebDriver e = getWebDriver();</span></span><br><span class="line">                    <span class="comment">// innerQueue.add(e);</span></span><br><span class="line">                    <span class="comment">// webDriverList.add(e);</span></span><br><span class="line">                &#125;</span><br><span class="line">            &#125;</span><br><span class="line"></span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">return</span> innerQueue.take();</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">returnToPool</span><span class="params">(WebDriver webDriver)</span> </span>&#123;</span><br><span class="line">        checkRunning();</span><br><span class="line">        innerQueue.add(webDriver);</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">protected</span> <span class="keyword">void</span> <span class="title">checkRunning</span><span class="params">()</span> </span>&#123;</span><br><span class="line">        <span class="keyword">if</span> (!stat.compareAndSet(STAT_RUNNING, STAT_RUNNING)) &#123;</span><br><span class="line">            <span class="keyword">throw</span> <span class="keyword">new</span> IllegalStateException(<span class="string">"Already closed!"</span>);</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">public</span> <span class="keyword">void</span> <span class="title">closeAll</span><span class="params">()</span> </span>&#123;</span><br><span class="line">        <span class="keyword">boolean</span> b = stat.compareAndSet(STAT_RUNNING, STAT_CLODED);</span><br><span class="line">        <span class="keyword">if</span> (!b) &#123;</span><br><span class="line">            <span class="keyword">throw</span> <span class="keyword">new</span> IllegalStateException(<span class="string">"Already closed!"</span>);</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">for</span> (WebDriver webDriver : webDriverList) &#123;</span><br><span class="line">            logger.info(<span class="string">"Quit webDriver"</span> + webDriver);</span><br><span class="line">            webDriver.quit();</span><br><span class="line">            webDriver = <span class="keyword">null</span>;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>以上代码参考自 <a href="https://github.com/code4craft/webmagic" rel="external nofollow" target="_blank">https://github.com/code4craft/webmagic</a></p><p>到此，动态加载的页面数据抓取就实现了。本文使用 selenium 作为渲染的方法，还有很多其他的方法，例如 phantomjs 和 htmlunit 等。有空了可以尝试其他的方法，希望本文对你有所帮助。</p><p><br>本文地址 <a href="http://ihongqiqu.com/2016/11/07/web-crawler-2/">http://ihongqiqu.com/2016/11/07/web-crawler-2/</a> 作者为<a href="/about/"> Zhenguo</a></p></div><div style="font-size:12px;BORDER-BOTTOM:#bbb 1px solid;BORDER-LEFT:#bbb 1px solid;BACKGROUND:#f6f6f6;MIN-HEIGHT:120px;BORDER-TOP:#bbb 1px solid;BORDER-RIGHT:#bbb 1px solid;MARGIN-LEFT:10px;MARGIN-RIGHT:10px;MARGIN-BOTTOM:10px" class="oec2003right"><div style="MARGIN-TOP:10px;FLOAT:left;MARGIN-LEFT:10px;MARGIN-RIGHT:10px"><img alt="author:Zhenguo" src="http://ihongqiqu.com/imgs/myself.jpg" width="100" height="100"></div><div style="LINE-HEIGHT:200%;MARGIN:10px;COLOR:#000">Author: <a href="http://ihongqiqu.com/">Zhenguo</a> &nbsp&nbsp&nbsp&nbsp Blog: <a href="http://ihongqiqu.com/">http://ihongqiqu.com/</a>&nbsp&nbsp&nbsp&nbsp Email: <a href="mailto:jinzhenguo1990@gmail.com" rel="external nofollow" target="_blank">jinzhenguo1990@gmail.com</a><br>I have almost 5 years of Android application development experience and have a keen interested in the latest emerging technologies. I use my spare time to turn my experience, ideas and love for Android tech into informative articles, tutorials and more in hope to help others and learn more.</div></div><footer class="article-footer clearfix"><div class="article-catetags"><div class="article-categories"><span></span> <a class="article-category-link" href="/categories/网络爬虫/">网络爬虫</a></div><div class="article-tags"><span></span> <a href="/tags/网络爬虫/">网络爬虫</a></div></div><div class="article-share" id="share"><div data-url="http://ihongqiqu.com/2016/11/07/web-crawler-2/" data-title="动态加载的页面数据的抓取 | 爱红旗渠" data-tsina="undefined" class="share clearfix"></div></div></footer></article><nav class="article-nav clearfix"><div class="prev"><a href="/2016/11/09/hexo-add-widget/" title="Hexo 自定义 widget"><strong>上一篇：</strong><br><span>Hexo 自定义 widget</span></a></div><div class="next"><a href="/2016/11/01/behavior-interview/" title="行为面谈"><strong>下一篇：</strong><br><span>行为面谈</span></a></div></nav><section id="comments" class="comment"><div class="ds-thread" data-thread-key="2016/11/07/web-crawler-2/" data-title="动态加载的页面数据的抓取" data-url="http://ihongqiqu.com/2016/11/07/web-crawler-2/"></div></section></div><div class="openaside"><a class="navbutton" href="#" title="显示侧边栏"></a></div><div id="toc" class="toc-aside"><strong class="toc-title">文章目录</strong><ol class="toc"><li class="toc-item toc-level-3"><a class="toc-link" href="#u5206_u6790"><span class="toc-number">1.</span> <span class="toc-text">分析</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#u5B9E_u73B0"><span class="toc-number">2.</span> <span class="toc-text">实现</span></a><ol class="toc-child"><li class="toc-item toc-level-4"><a class="toc-link" href="#u6DFB_u52A0_u4F9D_u8D56"><span class="toc-number">2.1.</span> <span class="toc-text">添加依赖</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u4E0B_u8F7D_driver"><span class="toc-number">2.2.</span> <span class="toc-text">下载 driver</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u5B89_u88C5_Chrome__u6D4F_u89C8_u5668"><span class="toc-number">2.3.</span> <span class="toc-text">安装 Chrome 浏览器</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u6D4B_u8BD5"><span class="toc-number">2.4.</span> <span class="toc-text">测试</span></a></li><li class="toc-item toc-level-4"><a class="toc-link" href="#u5B9E_u73B0_u722C_u866B"><span class="toc-number">2.5.</span> <span class="toc-text">实现爬虫</span></a></li></ol></li></ol></div><div id="asidepart"><div class="closeaside"><a class="closebutton" href="#" title="隐藏侧边栏"></a></div><aside class="clearfix"><div class="github-card"><p class="asidetitle">Github 名片</p><div class="github-card" data-github="jingle1267" data-width="220" data-height="119" data-theme="medium"><script type="text/javascript" src="//cdn.jsdelivr.net/github-cards/latest/widget.js"></script></div></div><div class="categorieslist"><p class="asidetitle">分类</p><ul><li><a href="/categories/Android-DataBinding/" title="Android DataBinding">Android DataBinding<sup>3</sup></a></li><li><a href="/categories/Android不规则图形专题/" title="Android不规则图形专题">Android不规则图形专题<sup>4</sup></a></li><li><a href="/categories/Android打多渠道包/" title="Android打多渠道包">Android打多渠道包<sup>3</sup></a></li><li><a href="/categories/Android测试/" title="Android测试">Android测试<sup>4</sup></a></li><li><a href="/categories/H5唤起原生应用/" title="H5唤起原生应用">H5唤起原生应用<sup>2</sup></a></li><li><a href="/categories/Seo优化/" title="Seo优化">Seo优化<sup>1</sup></a></li><li><a href="/categories/hexo/" title="hexo">hexo<sup>1</sup></a></li><li><a href="/categories/网络爬虫/" title="网络爬虫">网络爬虫<sup>2</sup></a></li><li><a href="/categories/设计模式/" title="设计模式">设计模式<sup>24</sup></a></li><li><a href="/categories/需求/" title="需求">需求<sup>1</sup></a></li></ul></div><div class="tagcloudlist"><p class="asidetitle">标签云</p><div class="tagcloudlist clearfix"><a href="/tags/Android-DataBinding/" style="font-size:16.67px">Android DataBinding</a> <a href="/tags/Android优化/" style="font-size:10px">Android优化</a> <a href="/tags/App-Links/" style="font-size:10px">App Links</a> <a href="/tags/Git/" style="font-size:10px">Git</a> <a href="/tags/IOC/" style="font-size:10px">IOC</a> <a href="/tags/Log/" style="font-size:10px">Log</a> <a href="/tags/Monkey/" style="font-size:10px">Monkey</a> <a href="/tags/Seo优化/" style="font-size:10px">Seo优化</a> <a href="/tags/cnblogs/" style="font-size:10px">cnblogs</a> <a href="/tags/hexo/" style="font-size:10px">hexo</a> <a href="/tags/instrumentation/" style="font-size:10px">instrumentation</a> <a href="/tags/monkeyrunner/" style="font-size:10px">monkeyrunner</a> <a href="/tags/唤起/" style="font-size:10px">唤起</a> <a href="/tags/开会/" style="font-size:10px">开会</a> <a href="/tags/影响力/" style="font-size:10px">影响力</a> <a href="/tags/心理学/" style="font-size:10px">心理学</a> <a href="/tags/打包/" style="font-size:16.67px">打包</a> <a href="/tags/收集需求/" style="font-size:10px">收集需求</a> <a href="/tags/敏捷/" style="font-size:10px">敏捷</a> <a href="/tags/权限/" style="font-size:10px">权限</a> <a href="/tags/网络爬虫/" style="font-size:13.33px">网络爬虫</a> <a href="/tags/行为面谈/" style="font-size:10px">行为面谈</a> <a href="/tags/设计模式/" style="font-size:20px">设计模式</a> <a href="/tags/说服他人/" style="font-size:10px">说服他人</a></div></div><div class="archiveslist"><p class="asidetitle"><a href="/archives">归档</a></p><ul class="archive-list"><li class="archive-list-item"><a class="archive-list-link" href="/archives/2016/11/">十一月 2016</a><span class="archive-list-count">3</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2016/10/">十月 2016</a><span class="archive-list-count">7</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2016/09/">九月 2016</a><span class="archive-list-count">8</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2016/05/">五月 2016</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2016/04/">四月 2016</a><span class="archive-list-count">2</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2016/03/">三月 2016</a><span class="archive-list-count">4</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2016/01/">一月 2016</a><span class="archive-list-count">3</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2015/12/">十二月 2015</a><span class="archive-list-count">6</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2015/07/">七月 2015</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2015/05/">五月 2015</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2014/12/">十二月 2014</a><span class="archive-list-count">7</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2014/11/">十一月 2014</a><span class="archive-list-count">12</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2014/10/">十月 2014</a><span class="archive-list-count">6</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2014/09/">九月 2014</a><span class="archive-list-count">4</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2014/08/">八月 2014</a><span class="archive-list-count">1</span></li><li class="archive-list-item"><a class="archive-list-link" href="/archives/2014/07/">七月 2014</a><span class="archive-list-count">5</span></li></ul></div><div class="hotarticles"><p class="asidetitle">热评文章</p><div class="ds-top-threads" data-range="monthly" data-num-items="5"></div></div><div class="linkslist"><p class="asidetitle">友情链接</p><ul><li><a href="http://www.cnblogs.com/jingle1267/" target="_blank" title="博客园爱红旗渠">博客园</a></li><li><a href="http://blog.csdn.net/m925986445" target="_blank" title="CSDN爱红旗渠">CSDN博客</a></li><li><a href="http://www.hostodo.cn/" target="_blank" title="年付低至10美元洛杉矶VPS主机" rel="external nofollow">Hostodo</a></li><li><a href="http://www.80iter.com/" target="_blank" title="八零IT人" rel="external nofollow">八零IT人</a></li><li><a href="http://salogs.com/" target="_blank" title="拥抱开源，分享经验!" rel="external nofollow">SA-Logs</a></li></ul></div><div class="rsspart"><a href="/atom.xml" target="_blank" title="rss">RSS 订阅</a></div></aside></div></div><footer><div id="footer"><div class="line"><span></span><div class="author"></div></div><section class="info"><p>生活不止眼前的苟且<br>还有诗和远方的田野</p></section><div class="social-font"><a href="https://github.com/jingle1267" target="_blank" class="icon-github" title="github" rel="external nofollow"></a> <a href="mailto:jingle1267@163.com" target="_blank" class="icon-email" title="Email Me" rel="external nofollow"></a></div><p class="copyright">Copyright ©2014-2016 <a href="/about" target="_blank" title="Jin">Jin</a></p></div><script type="text/javascript">var cnzz_protocol="https:"==document.location.protocol?" https://":" http://";document.write(unescape("%3Cspan style='display:none;' id='cnzz_stat_icon_1260545070'%3E%3C/span%3E%3Cscript src='"+cnzz_protocol+"s95.cnzz.com/z_stat.php%3Fid%3D1260545070' type='text/javascript'%3E%3C/script%3E"))</script><script type="text/javascript">!function(t,e,n,s,c,i,a){t.SwiftypeObject=c,t[c]=t[c]||function(){(t[c].q=t[c].q||[]).push(arguments)},i=e.createElement(n),a=e.getElementsByTagName(n)[0],i.async=1,i.src=s,a.parentNode.insertBefore(i,a)}(window,document,"script","//s.swiftypecdn.com/install/v2/st.js","_st"),_st("install","qFtymqNnfXjgmwxAJxMP","2.0.0")</script></footer><script src="/js/jquery-2.0.3.min.js"></script><script src="/js/jquery.imagesloaded.min.js"></script><script src="/js/gallery.js"></script><script src="/js/jquery.qrcode-0.12.0.min.js"></script><script type="text/javascript">$(document).ready(function(){function e(){"number"==typeof window.innerWidth?n=window.innerWidth:document.documentElement&&document.documentElement.clientWidth&&(n=document.documentElement.clientWidth)}$(".navbar").click(function(){$("header nav").toggleClass("shownav")});var n=0,s=$("#main"),a=$("#asidepart"),o=$(".closeaside"),d=$(".openaside");o.click(function(){a.addClass("fadeOut").css("display","none"),d.css("display","block").addClass("fadeIn"),s.addClass("moveMain")}),d.click(function(){d.css("display","none").removeClass("beforeFadeIn"),a.css("display","block").removeClass("fadeOut").addClass("fadeIn"),s.removeClass("moveMain")}),$(window).scroll(function(){d.css("top",Math.max(80,260-$(this).scrollTop()))}),$(window).resize(function(){e(),n>=1024?$("header nav").removeClass("shownav"):(s.removeClass("moveMain"),a.css("display","block").removeClass("fadeOut"),d.css("display","none"),$("#toc.toc-aside").css("display","none"))})})</script><script type="text/javascript">$(document).ready(function(){var c=$(".article-content>iframe"),n=$(".article-content>embed"),o=($("#toc"),$("#toc.toc-aside")),e=$(".openaside"),i=$(".closeaside");c.length>0&&c.wrap('<div class="video-container" />'),n.length>0&&n.wrap('<div class="video-container" />'),i.click(function(){o.css("display","block").addClass("fadeIn")}),e.click(function(){o.css("display","none")}),$(window).scroll(function(){o.css("top",Math.max(140,320-$(this).scrollTop()))})})</script><script type="text/javascript">$(document).ready(function(){function e(){"number"==typeof window.innerWidth?n=window.innerWidth:document.documentElement&&document.documentElement.clientWidth&&(n=document.documentElement.clientWidth);var e=n>1024?200:100,t={render:"image",size:e,fill:"#2ca6cb",text:r,radius:.5,quiet:1},a=$(".article-share-qrcode").position();$(".hoverqrcode").empty().css("width",e).css("height",e).css("left",a.left-e/2+20).css("top",a.top-e-10).qrcode(t)}var t=$(".share"),r=t.attr("data-url"),a=encodeURIComponent(r),i=t.attr("data-title"),c=t.attr("data-tsina"),o=(t.attr("description"),['<div class="hoverqrcode clearfix"></div>','<a class="overlay" id="qrcode"></a>','<a href="https://www.facebook.com/sharer.php?u='+a+'" class="article-share-facebook" target="_blank" title="Facebook"></a>','<a href="https://twitter.com/intent/tweet?url='+a+'" class="article-share-twitter" target="_blank" title="Twitter"></a>','<a href="#qrcode" class="article-share-qrcode" title="微信"></a>','<a href="http://widget.renren.com/dialog/share?resourceUrl='+a+"&srcUrl="+a+"&title="+i+'" class="article-share-renren" target="_blank" title="人人"></a>','<a href="http://service.weibo.com/share/share.php?title='+i+"&url="+a+"&ralateUid="+c+'&searchPic=true&style=number" class="article-share-weibo" target="_blank" title="微博"></a>','<span title="Share to"></span>'].join(""));t.append(o),$(".hoverqrcode").hide();var n=0;$(window).resize(function(){$(".hoverqrcode").hide()}),$(".article-share-qrcode").click(function(){e(),$(".hoverqrcode").toggle()}),$(".article-share-qrcode").hover(function(){},function(){$(".hoverqrcode").hide()})})</script><script type="text/javascript">var duoshuoQuery={short_name:"ihongqiqu"};!function(){var e=document.createElement("script");e.type="text/javascript",e.async=!0,e.src="//static.duoshuo.com/embed.js",e.charset="UTF-8",(document.getElementsByTagName("head")[0]||document.getElementsByTagName("body")[0]).appendChild(e)}()</script><link rel="stylesheet" href="/fancybox/jquery.fancybox.css" media="screen" type="text/css"><script src="/fancybox/jquery.fancybox.pack.js"></script><script type="text/javascript">$(document).ready(function(){$(".article-content").each(function(a){$(this).find("img").each(function(){if(!$(this).parent().hasClass("fancybox")){var a=this.alt;a&&$(this).after('<span class="caption">'+a+"</span>"),$(this).wrap('<a href="'+this.src+'" title="'+a+'" class="fancybox"></a>')}}),$(this).find(".fancybox").each(function(){$(this).attr("rel","article"+a)})}),$.fancybox&&$(".fancybox").fancybox()})</script><script>var _hmt=_hmt||[];!function(){var e=document.createElement("script");e.src="//hm.baidu.com/hm.js?9223f49a80da70c70261ac4e2f74af9a";var a=document.getElementsByTagName("script")[0];a.parentNode.insertBefore(e,a)}()</script><script type="text/javascript">var cnzz_protocol="https:"==document.location.protocol?" https://":" http://";document.write(unescape("%3Cspan id='cnzz_stat_icon_null'%3E%3C/span%3E%3Cscript src='"+cnzz_protocol+"s23.cnzz.com/z_stat.php%3Fid%3Dnull' type='text/javascript'%3E%3C/script%3E"))</script><div id="totop"><a title="返回顶部"><img alt="返回顶部" src="/img/scrollup.png"></a></div><script src="/js/totop.js"></script></body><!-- rebuild by neat -->